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ABSTRACT 

Findings of a study that determined the ways in which 
practicing administrative course developers utilize instructional 
design activities in their design projects are presented in this 
paper. A survey was administered to 30 participants in a training and 
development group within a large organization and mailed to 43 other 
practicing instructional designers from a wide variety of contexts. 
Approximately 47 percent of the mailed surveys were returned. The 
results of the study were: (1) the most frequently completed design 

activities were writing learning objectives, developing test items, 
selecting instructional strategies, and selecting members; (2) the 
least frequently completed design activities were assessing trainee's 
entry skills and characteristics, and conducting a pilot test; (3) 
the most frequently given reasons for not including a design activity 
were the decision was already made, there wasn't enough time, and it 
was considered unnecessary; (4) lack of instructional design was 
seldom identified as a reason for excluding a design activity; (5) 
years of experience did not appear to be related to how often course 
developers use various instructional design activities; and (6) while 
virtually all participants indicated that they usually or always 
wrote learning objectives, only two-thirds indicated that they 
usually or always conducted a task analysis. A conclusion is that, in 
practice, instructional design activities occur on an irregular 
basis. An instructional development model to guide decision making in 
a real-life context is needed. Three figures, three tables, and a 
copy of the survey are included. (Contains 7 references.) (LMl) 
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For a number of years there has been a continuing debate on 
the degree to which instructional design models are used in 
practice. A preliminary study was conducted to determine if 
and how practicing course developers include instructional 
design activities (e.g., conducting needs assessment, writing 
learning objectives) in their design projects. A survey was 
distributed to 73 practicing instructional designers. They 
were asked to indicate how often they used eleven 
instructional design activities and, as appropriate, to indicate 
why an activity was excluded from some projects. 



The results of the study can be summarized as follows: 

• the most frequently completed design activities 
were: 

- writing learning objectives 

- developing test items 

- selecting instructional strategies 

- selecting media 

• the least frequently completed design activities 
were: 

- assessing trainee's entry skills and characteristics 

- conducting a pilot test 

• the most frequently given reasons for not including 
a design activity were: 

- decision already made 

- not enough time 

- considered unnecessary 

• "lack (ID) expertise" was seldom identified as a 
reason for excluding a design activity 

• years of experience did not appear to be related to 
how often course developers use of various 
instructional design activities 
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Findings (cont.) 



Conclusions 
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Interestingly, while virtually all participants indicated that 
they usually or always wrote learning objectives, only 
two-thirds indicated that they usually or always conducted 
a task analysis. This finding raises questions about how 
learning objectives are specified. 



While preliminary in nature, the findings provide 
evidence that, in practice, instructional design activities 
occur on an irregular basis, with some activities almost 
always occurring and other activities occurring perhaps 
half the time. This unevenness is problematic in that the 
typical instructional development model is based on the 
premise that all activities which comprise the model will 
be completed. Indeed, some models essentially require 
that certain activities be completed before other activities 
are undertaken. The prescriptions of such models appear 
to be regularly violated in practice, underscoring the need 
for an instructional development model which can guide 
decision making in a real-life context. 

Additional research is needed to understand the nature of 
the tasks and approaches used by practicing instructional 
designers. The validity and reliability of the survey 
instrument needs to be established and used with a large 
sample. Additional demographic data needs to be 
collected and analyzed. 

Indepth interviews and observational studies with 
instructional designers should also be conducted. Such 
efforts will help determine the decision-making factors 
and principles which drive the practice of instructional 
design. 
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What do you do 

When you have no time 

No time to do "model" ID? 

Do you skip some steps. 

Or water them down. 

Combine, or maybe all three? 

The Practitioner’s Lament 
circa 1989 



The Practitioner's Lament raises several questions: 

• Which steps are most likely to be skipped, 
combined, or diluted? 

• Is time actually an obstacle in completing course 
development projects? 

• Are there other factors which are also obstacles to 
implementing ID models? 

This study was conducted to provide preliminary answers 
to these questions. 
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Background 



Criticism of ID/D Instructional design and development (ID/D) models are 

Models used as a guide to quality instruction. However, ID/D 

models in general have been criticized for their 
complexity and lack of adaptability. 

For example, Rogoff (1984) called for simpler, more 
pragmatic approaches than allowed for in most ID/D 
models. More recently, Rosenberg (1991) argued that ID/D 
models need to be more efficient. Rosenberg pointed out 
that present models ". . . lack the ability to know what to 
skip and what not to skip. Thus, the process takes too 
long and frustrates management" (p. 5). 



Layers of Necessity The Layers of Necessity Model (Tessmer & Wedman, 1990; 

Wedman & Tessmer, 1990) provides a flexible ID/D model 
that helps adapt design and development activities to 
project circumstances (see Figure 1). 

Figure 1. Layers of Necessity Model 
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Background (cont.) 



Layers of Necessity In effect, the model is an attempt to integrate numerous 

ID/D activities, which range in terms of complexity and 
precision, into a single framework. The model is 
intended to be sensitive to the load, payoff, and pressure 
factors that influence decisions to include, skip, or water- 
down various activities within a given project (Tessmer & 
Wedman, 1992). A partial listing of load, payoff, and 
pressure factors is provided in Table 1 below. 

Table 1. Sample Load, Payoff, and Pressure Factors 



Load Factors 


Payoff Factors 


Pressure Factors j 


Time 

Money 

Personnel Expertise 
Project Difficulty 


Performance Impact 
Client Satisfaction 
Designer Status 
Organizational Image 


Norms 
Tolerance 
Client Demand 
Resource Use 



A combination of load, payoff, and pressure factors are 
hypothesized to be at work in any ID/D project. 
Collectively, these factors can influence the decision to 
conduct a given design activity and can also influence the 
extent to which the activity is carried out. 

Consider, for example, two course development projects. 
One project has a very short timeline, a client who views 
instruction as a necessary evil, and a limited budget. The 
other project has ample time, a client who is dedicated to 
quality instruction, and an substantial budget. The first 
project would be a likely candidate for a streamlined ID/D 
approach; the second project might allow for a more 
sophisticated approach. Each of these are briefly discussed 
below. 
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Background (cont.) 



The first project calls for a simple layer of ID/D activities 
(see Figure 2). None of the activities would be carried out 
in detail, and careful attention would be given to limit the 
time and resource consumption of the activities. For 
example, the "situational assessment" activity might be 
limited to answering the following questions: 

• What performance improvement is required? 

• Will instruction improve performance? 

• Who will be receive the instruction? 

• What are the resource and time constraints? 

Similar limitations would be placed on the other ID/D 
activities. 

Figure 2. Simple layer of ID activities 
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Background (cont.) 



In the second example, additional time, resources, and 
support were available, allowing additional activities to be 
included (see Figure 3 below). 

Figure 3. Multiple layers of ID activities 




Is there evidence to support the need for a layered approach? 
Do practitioners skip ID/D activities? If so, what are the 
factors which influence the decision to skip an activity? This 
paper provides a preliminary answer to these questions. 
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The subjects in the study consisted of 73 practicing 
instructional designers in two states. The subjects' 
experience in training ranged from a few months to over 
25 years. 

The subject's job responsibilities covered many facets of 
training and development, including course writing, 
project management, and stand-up training. For some, 
training and development was their career; for others, 
their assignment to training was temporary. 

Approximately 40% of the subjects (N=30) were from the 
same training and development group within a large 
organization. The other subjects (N=43) were from a wide 
variety of contexts (e.g., business, government). 



An "Instructional Design Activities Survey" was 
constructed for the study. The survey was comprised of 
two parts. The first part asked the subjects to rank the 
frequency with which they completed each of eleven 
design activities on instructional design projects. The 
design activities list was a composite of several common 
ID/D models (e.g., Dick & Carey, 1985). The eleven design 
activities were: 

• conduct a needs assessment 

• determine if need can be solved by training 

• write learning objectives 

• conduct task analyses 

• identify the types of learning outcomes (concepts, 
prob. solving) 

• assess trainee's entry skills and characteristics 

• develop test items 

• select instructional strategies for training 

• select media formats for the training 

• pilot test instruction before completion 

• do a follow-up evaluation of the training 




Methodology (corn.) 
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Instrument (cont.) 



Data Collection 
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Frequency was expressed in terms of Never, Occasionally, 
Usually, and Always. The instrument also provided space 
for the participants to add an additional design activity not 
listed on the survey. 

The second part of the survey asked the subjects to pick 
one or more reasons to explain why each design activity 
might be excluded from some projects. The six options 
were: 

• lack expertise 

• client won't support 

• decision already made 

• considered unnecessary 

• not enough time 

• not enough money 

Space was also provided for the subjects to add an 
additional reason why a design activity was excluded from 
some projects. 



Two data collection procedures were used. The survey 
was distributed to the intact training and development 
group during a workshop being conducted for these 
individuals. (Note: The workshop did not deal with 
instructional design issues). For the remaining subjects, 
the instrument was distributed by mail using a mailing 
list provided by a local professional association. 
Approximately 47% of the mail-out surveys were 
completed and returned. 
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Analysis 



As mentioned earlier, approximately 40% of the subjects 
were from the same training and development 
organization; the remaining subjects were from a variety 
of contexts. An analysis (Mann- Whitney U) was 
performed on an item-by-item basis to determine if the 
data from the in-tact group could be combined with the 
data from the remaining subjects. Since no statistically 
significant differences were found between the two 
groups' responses (the level of significance hovered 
around .49 on each item), the data were aggregated. 

An early hypothesis was that experience would play a role 
in designers' decision to include ID/D activities. To test 
this hypothesis, the responses were grouped according to 
years of experience and analyzed (Mann-Whitney U) to 
determine if years of experience was related to the 
decision to complete an activity. 

Frequencies and percentages were computed for the level 
of use reported for each design activity. Frequencies were 
also computed for the reasons why an activity was 
excluded from some design projects. The frequencies and 
percentages were examined for general trends. 

Very few subjects exercised the option of adding an 
additional design activity or an additional reason for 
excluding a design activity. Consequently, these data were 
not included in the analysis. 



Findings 
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ID/D Activity The most frequently completed ID/D activities were "write 

Completion learning objectives," "select instructional strategies," and 

"select media formats." The least frequently completed 
activities were "pilot test" and "assess trainee's entry skills." 

To gain a broader perspective on the likelihood an activity 
would be completed, two composite values were calculated. 
The "Probably Will" value is the sum of the "Always" and 
"Usually" percentages; the "Probably Won't" value is the 
sum of the "Occasionally" and "Never" percentages. Table 2 
below summarizes these percentages. Percentages 
warranting special attention are underlined. 

Note. The numbers in the cells represent the percentage of the total 
number of subjects who marked a particular category. The 
percentages will not total to 100% since some subjects did not 
respond to every item. 



Table 2. Percentages for Design Activity Completion 





How ofti 


Bn an activity is completed 


ID/D Activity 


Always 


Usually 


Probably 

Will 


Occas'ly 


Never 


Probably 

Won't 


Conduct a needs assessment 


22 


34 


63 


23 


8 


31 


Determine if need can be 
solved by training 


36 


34 


70 


21 


8 | 


29 


Write learning objectives 


82 


12 


94 


4 


1 ; 


5 


Conduct task analyses 


j 30 


36 


66 


23 


8 


31 


Identify the types of learning 
outcomes 


36 


38 


74 


16 


10 


26 


Assess trainee's entry skills 
and characteristics 


27 


27 


54 


34 


io ; 


M 


Develop test items 


59 


23 


82 


11 


4 


15 


Select instructional strategies 
for training 


50 


35 


85 


10 


3 


13 


Select media formats for the 
training 


52 ; 


34 


86 


8 


3 


11 


Pilot test instruction before 
competition 


33 


16 


49 


33 


18 


51 


Do a follow-up evaluation of 
the training 


38 


34 


75 


22 


5 ! 


27 




14 
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Findings (cont.) 



Why an Activity Is The reasons given for why an activity was excluded from 

Excluded some projects ranged from "decision already made" 

(selected 130 times) to "lack expertise" (selected 22 times). 
Somewhat surprisingly, "not enough money" was not 
identified as a common reason for excluding an activity. 
Table 3 summarizes these findings on an activity-by- 
activity basis. Frequencies which warrant special attention 
are underlined. 

Note. The numbers in the cells represent the number of times a particular 
reason was selected. Subjects were asked to indicate all reasons 
that applied to a particular activity. 



Table 3. Reasons for Excluding ID/D Activity 





Reasor 


s an acl 


ivity is excluded 








ID/D Activity 


Lack 


Client 


Decision 


Consid. 


Not 


Not 


Total 


Expertise 


Won’t 

Support 


Already 

Made 


Unneces. 


Enough 

Time 


Enough 

Money 


(across 

reasons) 


Conduct a needs assessment 


0 


6 


31 


15 


13 


3 


68 


Determine if need can be 
solved by training 


3 


1 

2 




29 


5 


6 


1 


46 


Write learning objectives 


2 


0 


2 


5 


5 


3 


17 


Conduct task analyses 


5 


5 


12 


17 


14 


3 


56 


Identify the types of learning 
outcomes 


7 


7 


13 


12 


9 : 


2 


50 


Assess trainee’s entry skills 
and characteristics 


0 


6 


11 


16 


8 i 


4 


45 


Develop test items 


0 


2 


3 


12 


7 


3 


27 


Select instructional strategies 
for training 


3 


1 


12 


1 | 


9 


4 


30 


Select media formats for the 
training 


0 


2 


11 


2 


9 


9 


33 


Pilot test instruction before 
competition 


1 


9 


4 


15 


32 


14 


75 


Do a follow-up evaluation of 
the training 


1 


8 


2 


9 


9 j 


7 


36 


Total (across activities) 


22 


48 


130 


109 


121 


53 


— 
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Discussion 



Selective Completion 
of ID/D Activities 


The results of this preliminary study confirm that 
practicing instructional designers do not complete all 
design activities for all projects. While not particularly 
surprising, this finding does raise questions about the 
relationship between ID/D models and ID/D practice. The 
Layers of Necessity Model is being developed in an 
attempt to improve this relationship. 

However, several other trends can be seen in the data. 
These trends, and the ensuing questions, are discussed 
below. 


Objectives without 
Task Analysis 


The first trend is the relationship between certain ID/D 
activities. As noted above, the vast majority of subjects 
reported they always or usually wrote learning objectives. 
However, nearly one-third (31%) of the subjects indicated 
they were not likely to conduct a task analysis, raising 
questions about how objectives are derived. When 
coupled with the finding that the most frequently given 
reason for not conducting a task analysis was "considered 
unnecessary," questions about the rationality of ID/D 
practice begin to arise. 


Quality Control & 
Pilot Testing 


A second trend is in the area of quality control. The sole 
reason for pilot testing instruction is to test the quality of the 
instruction prior to full-scale implementation. Pilot testing 
is considered necessary because it is unlikely that "first draft 
instruction" will be without problems. However, in spite of 
the importance of pilot testing, only half of the subjects 
indicated they pilot tested instruction on a regular basis. 
"Note enough time" is the overwhelming reason given for 
not including this activity in a project. Does pilot testing 
takes too long, or do the other design activities take too 
much time? 



Discussion (cont.) 



Trainee Analysis 



Barriers to ID/D 
Activities 
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A third trend warranting discussion involves assessing 
trainee's entry skills and characteristics. Given the 
importance of prior knowledge (i.e., entry skills) and the 
popularity of "learning style" and "adult learner" 
constructs, one might expect some form of preassessment 
to be a common occurrence. Such is not the case; nearly 
half of the subjects indicated they were not likely to assess 
entry skills or characteristics. And the most frequently 
given reason for excluding this activity: "considered 
unnecessary." 



The final trend to be discussed here is not about any 
particular ID/D activity but rather about the reasons given 
for excluding an activity. As expected, "Not enough time" 
was identified as a common reason. 

However, "decision already made" and "considered 
unnecessary" was also cited as common reasons. In terms 
of the "decision already made" reason, several questions 
immediately surface: "Who is making these decisions?" 
"How are they being made?" "Are they actually being 
made or are 'default values' used?" In terms of the 
"considered unnecessary" reason, are certain activities 
inappropriate for some project, or are they considered to 
add little value to the process? 
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Conclusion 



This study was undertaken in an attempt to provide 
support for a Layers of Necessity type of approach to 
instructional design. The results did provide such 
support; ID/D activities are not always carried out and 
time is a frequently cited reason for excluding certain 
activities, especially pilot testing instruction. 

Somewhat surprisingly, the lack of money was not 
identified as a major barrier. While resource constraints 
will continue to be accounted for, future work on the 
Layers of Necessity Model will concentrate on identifying 
way to streamline the course development process. A 
means of "cutting corners while controlling risk" in ID/D 
project needs to be developed. 

Unfortunately, the implications of the study do not stop 
here. While preliminary and tentative, the results raise 
serious questions about the relationship between 
instructional design and development as it generally 
appears in the literature and as it is practiced in real life. 

• Perhaps the survey was flawed. 

• Perhaps the subjects were atypical. 

• Perhaps the interpretation was too sweeping. 

• Perhaps additional, better, and different research 
into the practice of instructional design and 
development is needed. 

• Perhaps we need to shift our attention from better 
instructional design models to better instructional 
design practice. 

Whatever the case, we need to end the "disconnect" 
between instructional design models and the realities 
these models must be able to address. 
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Campus Box 106 P.O. Box 173364 Denver, Colorado 3021 7-3364 
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Instructional Design 
Activities Survey 

Dear Fellow NSPI Member: 

For a number of years there has been a continuing debate on the degree to which 
instructional design models are used in practice. This survey, however simple, is our 
attempt to determine if and how performance technologists use instructional design 
activities in their design projects. 

We would like you to tell us which design activities you omit from projects in which they 
could be employed, and why. Please confine your answers to projects in which you 
have participated. The answers to this survey will be used to tell technologists how 
and why ID activities are used in practice. We hope to present the results at an 
upcoming NSPI meeting and in a forthcoming Performance and Instruction article. 

Please take a few minutes to complete this survey and return it to us in the enclosed 
envelope by November 27 . All survey responses will be confidential. 

Instructions 



PLEASE RETURN THIS SURVEY IN THE ENCLOSED POSTAGE-PAID ENVELOPE. If you have any questions , 
please feel free to call Marty Tessmer, University of Colorado at Denver (303) 556-4362. Thank you for your help 
in this study. 



Job Title 

Your full job title is: 



Job Duties 

Please describe your primary job duties. 



If you would be available for a short 

Name 



Your Position (Check One) 

Independent Training Consultant 

Member of an Independent Training 

Organization 

Member of a Training Dept, within a 

larger Organization 

Other: 



Experience 

How many years have you had training-related 
duties? 

-- Years 

followup phone call, please provide: 

Phone # 



(OVER) 



How Often Do You Do This On ID Projects? Why is This Activity Excluded 

(Check One Only) from Some Projects? 

(Check all that apply) 
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